Any help? Anyone else seeing crashes with the AddressBook preview?

Ahh, finally found the right answer here: https://github.com/iterate-ch/cyberduck/issues/16031 Basically, the default behavior of hiding the actual Mac's MAC address is the cause for the problem ("Privaty Wi-Fi Address"): If the Mac rotates or replaces the primary network interface's MAC address, and my code still uses the old code that would fetch the MAC from the higher level APIs, it would get the wrong MAC, and then fail the receipt check. To fix this, the MAC has to be fetched from IOKit registry, e.g. as shown in https://developer.apple.com/documentation/appstorereceipts/validating-receipts-on-the-device or https://lapcatsoftware.com/articles/2023/11/4.html

This is getting worse - a person who gets this error sent me the receipt file, and it turns out that the receipt file is fine, meaning that the validation shouldn't even fail, and thus the app not exit with 173. And since my app doesn't use any in-app purchases either, I'm now clueless what's actually causing this. In other words: What exactly causes this message to appear?

I can't seem to edit the message now that I know a bit more. So, here's a reply instead. Turns out that the basic receipt validation does not use SHA-256. The message rather seems to come up when my app quits with exit code 173 due to the receipt being invalid (as in when the app was copied from a different Mac, and now the GUID doesn't match). Whereas older macOS versions would then re-fetch a new receipt (and optionally ask the user to authenticate again), this doesn't apparently happen any more (WhyTF not? Why not keep the old method and only use the new method when the app makes it known to support the new method?!) Forcing a re-fetch of the receipt by simply deleting and re-installing the app from the MAS doesn't appear to work either as the receipt appears to be cached somewhere (again WTF - why isn't macOS being cautious here and falls back to downloading it when the app keeps returning 173???) It appears the app now has to explicitly request a new receipt. With SKReceiptRefreshRequest, which is already deprecated, but no alternative is offered (I don't do in-app purchases and also don't use Xcode/Swift but something different that doesn't offer access to these classes). Searching for the error message turned up other reports where people ran into this problem with old programs that cannot be launched any more just because of this receipt caching that apparently doesn't fix itself. I can't understand why macOS is handling this so badly. Is that just ignorance or thoughtlessness on behalf of the responsible dev team at Apple or is there some good reason why macOS can't give older apps and their long-time users a break here? Or me, who has to spend hours to keep the app working just to keep up with such changes (another one is that suddenly in Tahoe right-clicks into the menu bar item don't register any more if the mouse is at the top edge of the menu bat - which totally violates basic UI concepts established over 40 years ago). This year isn't going well so far for me.

To anyone filing bug reports: Please leave a copy at https://openradar.appspot.com/ so that others can see what the problem is you reported.

The issue was a symlink in my app: The Info.plist points to a file for the app icon. That file is a relative symlink to an existing icon file. This used to work for many years, and the Finder has no problem with it, either. Just Apple's recent verification process stumbles over it, without sending an error msg about it to me.

Specifically comparing searchfs vs getatrrlistbulk, if you end up having read the entire catalog structure off disk, then that cost can mask any performance difference. I just did a new comparison: I compared how searchfs compares vs. recursive dir scan on HFS+ vs. APFS. On SSDs, of course. Always searching for a non-existing file name (so that no extra attributes need to be collected). A search on a HFS+ vol with 8.8M items (6.6M files, 2.2M dirs) on a fast Intel system: searchfs: 40s find: 270s And on a HFS+ vol with 2.6M items (2M files, 600k dirs), Mac Mini 2018: searchfs: 11s find: 70s We can see that the search locally in the FS makes a huge different here. About factor 6 each time. Probably due to the many kernel-user space transitions. Now a search on APFS, with 1.75M items (1.5M files and 250k dirs), on an M4 Mini: searchfs: 13s find: 35s And APFS with 490k items (360k files, 130k dirs), on an Intel Mini 2018: searchfs: 9s find: 40s So, there's still a difference, but smaller. Which may suggest that there's still room for improvement on the searchfs part, or not. But it seems that on APFS everything (both searchfs and readdir etc) is significantly slower compared to HFS+, which is probably due to the dual-tree nature of APFS that requires more reads from the medium. With this in mind, the difference of searchfs performance of HFS+ vs. APFS becomes smaller, and so my bug report (radar 35052030, FB5367373) isn't really an issue. I've justed close it. I suggest we put this to rest. There's nothing to be done about this anyway. I'd rather have someone focus on how to make the Finder be faster when browsing the /Applications folder, e.g. by avoiding needless repeat calls into the FS, because that affects everyone, and to a point where one can get impatient (just try to mount another Mac's startup volume over SMB and browse that Mac's Apps folder over the network and it's time to make a coffee).

[quote='830819022, DTS Engineer, /thread/776198?answerId=830819022#830819022'] However, the real benefit here isn't local filesystem, it's remote file system case where the core cost isn't the direct syscall cost, but the cost of transferring all of those individual file records.[/quote] Right - but that huge advantage was lost when Apple decided to drop AFP support and didn't add such a feature to SMB. (And this is part of my current pain having to find alternative search methods (https://findanyfile.app/fs/) for various servers, such as Spotlight, SSH login with "find" execution on the server, using Everything's http server on Windows, and so on. It was all so easy when we had AFP). Is there even any supported network file system left that supports searchfs? Does NFS (I believe I never checked because it's so diffucult to use)?

See also: https://github.com/KhaosT/FSKitSample and https://developer.apple.com/forums/thread/765424

Wow, Kevin. That's excellent support, thank you so much for making the effort! I had no idea that using concurrency in file system calls would have any such effect. You proved me wrong. In a real implementation, I think you'd probably want a fixed set of threads calling "getattrlistbulk", another set of threads processing the data each call produced, and then system the pushed directory back to the syscall thread. Already doing all the processing past the initial gathering of the directory contents in a separate thread, but that's clearly far from what's possible. I actually tracked this bug down and I have a question about your testing. Were you always testing on "freshly" mounted volumes (meaning, umount volume-> mount volume-> searchfs)? Or did you also test on volumes under normal use and repeatedly running searchfs? The use case of my app is to mainly run searches on the entire startup volume. So that's what I focused on with my tests. I try to be aware of caching effects (I believe some OS versions can use a of the available memory for caching information, but never learned about the details - all info on this I found was fairly vague). What I did, for instance, was to run searchfs on a running High Sierra system using HFS+, then cloned the system and let it convert to APFS, booted from that and ran the same tests. Repeatedly, of course. Later ran the tests on upgraded macOS systems, showing no significant improvement. The thing that confuses me most about searchfs performance on APFS is that when I search for something that's not findable (e.g. a unique name) both with searchfs and with a conventional method that eventually goes thru getatrrlistbulk, it didn't make much of a difference - even though the number of syscalls should be immensely different (and thereby contributing a lot to the overall time): searchfs should be a single syscalls, whereas the other method will be in the 100000s, right? That is why it could be so fast on HFS+, because all the searching happened right in the VFS, in a tight loop. Am I misunderstanding how it's working on APFS? I always assumed it was because of the dual-tree nature of APFS, where one cannot simply look at the one tree containing the dir records, looking for matching names, but must also look up the corresponding entries in the other tree in order to figure out which volume the dir entries belong to, and this would be the bottleneck (though one could argue that simply first blindly looking for matches in the dir rec tree and only then verifying the other tree if a match is found may be faster than the other way around that I suspect it's happening now). You understand what I mean? searchfs may be optimized to either first sort out the right dir recs by making sure they belong to the target volume before checking the dir properties, or vice versa, and maybe the other way around would be usually faster (assuming one seeks to find a small set of hits). Of course, that would require more work on this code for very few gain (hardly anyone is using this anyway, I believe). FYI, "find" is opensource and seems to be a relatively straightforward implementation built on fts. Yes, I know, and still my fts code in DirScanner was slower. So I assumed it had to do with the use of NSURLs. However, I must now admit that I was wrong in that. While find is still twice as fast compared to the search performance in FindAnyFile, it is now on-par with my most recent DirScanner version: After Jim Luther told me about the key to using fileSystemRepresentation instead of converting paths to NSStrings in my code, DirScanner became significantly faster. Yet, I failed to contemplate that fact when I wrote above that fts would still be 2x faster, or was too lazy (or confident) to double check. Now, with your findings on running parallel dir reads, I have a big task ahead of me rewriting that code. I'm excited. Thank you again for all your input.

Kevin, I appreciate your involvement in this, and your willingness to dive deeper into this for my (and hopefully others') education Some of the things you explain about performance were already clear to me (I used to write file systems and a simple operating system in the long past). Some were not clear, e.g. I have assumed that maybe some of the NS based calls were happening in a kernel space where making calls to the lower APIs would be less wasteful, or that NSURLs, which these calls return would be pooled at a more efficient level (where I only recently learned from Jim L that it was for me using NSString for paths where the fileSystemRep was the key to keeping it faster). I also agree that some of the properties, such as "NSURLIsVolumeKey" shouldn't be costly because they should not require another call into the BSD/POSIX APIs as I'd have thought that the information is already present at the level above the VFS. And that's especially true for the NSURLIsDirectoryKey. Architecturally, the performance here should basically be "identical", as the ONLY reason "hasDirectoryPath" can work at all is that the enumeratorAtURL "told" the URL it was a directory at the point it was created Exactly my thought - I concluded that from logic, without looking at the source code. You're aware that NSURLs append a "/" suffix to dirs, while paths do not, right? So, I suspect that hasDirectoryPath simply checks for that trailing "/", hence it being so fast. BTW: If you time the "find" tool, you'll see that it can be almost twice as fast as my DirScanner tests. Which surprised me. I still wonder if NSStrings conversions play a role in that, which the find tool doesn't have to worry about. [quote='830058022, DTS Engineer, /thread/776198?answerId=830058022#830058022']the only explanation I have is that there's a serious issue in our current implementation and the resource cache simply isn't working properly. Please file a bug on this and send me the bug number once it's filed.[/quote] Huh, well, okay. I was already expecting that anyway, but never bothered to report it as a problem because I didn't think anyone would bother with it. I had reported, for instance, an issue with searchfs(), which has become much much slower in APFS over HFS+ (like, 6 times slower for the same amount of files, when it's an entire startup volume), and this was not addressed, as far as I can see. And if we're talking about performance issues on the FS, the worst offender is Finder and its support frameworks. I have once run a trace of calls (e.g. with fs_usage) and I found that there's a LOT of repeat calls for the same attributes, which explains why nowadays browsing the Applications folder, especially over a network volume, is so awfully slow (it's mainly about diving into the bundles). THAT needs addressing, and it's so obviously bad - it should be obvious to anyone using a Mac for a few hours, especially if they have some experience how fast it should be (and indeed used to be!). As long as that isn't tackled, I wonder why I should make an effort to report these rather minor bottlenecks. But then again, maybe, if enumeratorAtURL is used by Finder a lot (can't dtrace anymore, can we?) and and this gets improved, maybe it'll automagically also improve browsing in Finder all over, significantly? If you think so, I'm happy to make the effort. (Feel free to email me directly if you have private comments, I don't want to start a flame war here, but rather really care about this getting better, if I can make a difference. Incidentally, I had applied for a job a few times working on FS improvement in the past but nothing come of it.) [quote='830058022, DTS Engineer, /thread/776198?answerId=830058022#830058022'] my advice would probably be to build and optimize entirely on getattrlistbulk. [/quote] Yes, that's indeed on my wish list, after seeing how the find tool can be so much faster (my app has now a mode where it calls the find tool and then my app just looks up the paths it reports back, and that's twice as fast). But then, I also want to improve the UI a lot and add a caching of the FS by tracking all changes and keeping a shadow directory, and that'll benefit my users more in the long run). Only so much a single programmer can accomplish :) using multiple threads and making multiple calls into getattrlistbulk simultaneously Yes, that's on my list as well, and yes, especially for network vols. Also, I had already planned this before SSDs became common, and I'd thought to identify which drives where on the same HDD, and only run concurrent searches on separate HDDs, to avoid excessive seeking. Though, I still have to experiment with this, because I'm not sure if the VFS would queue calls on a global level - but in recent years I got hints that this would not be the case, and I'd be able to have multiple FS calls run concurrently if they're on independent file systems, and you seem to indicate the same. Which is promising. However, most users of my app are simple "home" users who only search on their startup disk, and there's little to parallellize there. I had, for instance, considered to pre-cache the locked system volume, but then, the time for searching that volume is fairly quick compared to the /System/Volumes/Data volume, so caching the former doesn't gain much in the overall search time once the user has accumulated lots of files (which happens eventually). Again, thank you for not only trying to answer my questions but also exploring solutions.

Kevin, I'm using both a simple recursive iteration (contentsOfDirectoryAtURL) and enumeratorAtURL. In both cases, before I read the dir contents, I check if it's a volume with a trigger flag - if it is, I call opendir, readdir, closedir before I invoke one of the above FSFileManager methods again. That way, at least in my use case, the sub vols get mounted right away by the opendir/readdir, and the next filemanager call provides me with the new volume's contents. I only have one user out of 10000s who has reported this issue so far, for the >15 years I have my app in the public, so I consider this a "quick'n dirty" fix that works well enough for now. I've noted down your comments, though, in case I have to work on this again. I have a concern about performance, though: I like to be as fast as possible with the iteration, especially if I only need to look at file names. So, when calling enumeratorAtURL, I fetch no extra URL properties, and detect directories by called url.hasDirectoryPath, which turns out to be faster than checking NSURLIsDirectoryKe. If it's a dir, then I call getattrlist to get the mountpoint values. (See my "DirScanner" test project at https://files.tempel.org/Various/DirScanner.zip, which also employs this technique to avoid diving into other volumes, e.g. when browsing from "/" down.) Any idea if there's a better way? (I also learned recently from Jim Luther that it's much faster to use the C-string "url.fileSystemRepresentation" path instead of first getting url.path and then getting its UTF8String because that skips come costly internal conversions. Same for creating URLs when the name/path comes from POSIX/BSD functions. I've updated my DirScanner code accordingly.)

Ironically, the answer is found in a 10 year old post by myself (https://stackoverflow.com/a/28663838/43615). Basically: Before diving into the dir, get the ATTR_DIR_MOUNTSTATUS attribute and check the DIR_MNTSTATUS_TRIGGER flag. If it's set, it means the dir is currently not mounted but will trigger a mount once it gets read. So, if the flag is set, I only have to read the dir's contents (which will report back as being empty), and then I can immediately read again, and this time the mount will be there. I thought that there would be a delay, but apparently this happens synchronously. Alternatively, if I do not want to check the flag beforehand, I could also, if I find an empty dir, check if the dir is a mountpoint (DIR_MNTSTATUS_MNTPOINT flag) and if it is, re-read the dir.

I am in the same boat and am now confused - I thought the solution, for ObjC, was to use SKReceiptRefreshRequest to refresh the receipt?! My personal problem, though, is that I cannot test my code now. I seem to need a fresh system that has never seen the app before. I tried to create a VM with Fusion 11.5.3 as well as 13.5.2, but in all cases, I was only able to install macOS 10.14 or 11, but then upgrading to macOS 13 or 14 always results in kernel panics during install or shortly thereafter, with all settings still being defaults (other than increasing CPU cores and memory). Of course, this needs to be done on an Intel CPU Mac, because the Silicon Macs don't support the App Store in VMs. This is so fooked up.

Done: FB15568200